Methods and systems for automated selection of regions of an image for secondary finishing and generation of mask image of same

ABSTRACT

Automated systems, methods and tools that automatically extract and select portions of an image to automatically generate a premium finish mask specific to the image which require little or no human intervention are presented. Graphical user interface tools allowing a user to provide an image and to indicate regions of the image for application of premium finish are also presented.

FIELD OF THE INVENTION

The present invention relates to designing and manufacturing productshaving first surface areas having a first type of finish applied andsecond surface areas having a second type of finish applied, and moreparticularly, to the methods, tools, and systems for designating areasof different types of finish in a design to be applied to a productbased on an image of the design, and further to automated generation ofa corresponding mask image associated with the design.

BACKGROUND OF THE INVENTION

Printing services Web sites allowing a user to access the site from theuser's home or work and design a personalized product are well known andwidely used by many consumers, professionals, and businesses. Forexample, Vistaprint markets a variety of printed products, such asbusiness cards, postcards, brochures, holiday cards, announcements, andinvitations, online through the site www.vistaprint.com. Printingservices web sites often allow the user to review thumbnail images of anumber of customizable design templates prepared by the site operatorhaving a variety of different styles, formats, backgrounds, colorschemes, fonts and graphics from which the user may choose. When theuser has selected a specific product design template to customize, thesites typically provide online tools allowing the user to incorporatethe user's personal information and content into the selected templateto create a custom design. When the design is completed to the user'ssatisfaction, the user can place an order through the web site forproduction and delivery of a desired quantity of a product incorporatingthe corresponding customized design.

Finishes such as foil, gloss, raised print, vinyl, embossment, leather,cloth, and other textured finishes (hereinafter a “secondary finish”)that must be applied to a printed product separately from thetraditional ink application are typically reserved only for premiumprinted products due to the expense, time, and equipment required fordesign, setup, and application of the premium finishes. To add a premiumfinish to a printed product, at least one premium finish mask,designating areas where the secondary finish is to be applied versusareas where the secondary finish is not to be applied, must begenerated. The generation of the mask requires knowledge of whichportions of the design are to be finished using the secondary finish(i.e., foiled, glossed, raised print, or other premium finishes).

Currently, there are no tools for automatically extracting regions of adesign designated for application of a premium finish (e.g., foil,gloss, raised print, etc.) which would produce satisfactory results inmost cases which corresponds well to human judgment. In the printingworld, premium finishes are generally used to accent or highlightfeatures of an image. Generally, the determination of which featureslook good when accentuated using a premium finish is better left to ahuman designer because humans can more easily understand the meaning andrelationship between the different areas of the image and can quicklyselect those areas of an image that make sense (to other humans) tohighlight and to disregard for highlighting those areas that woulddetract from the overall aesthetics of the final product. It isdifficult to translate this type of human judgment into a computeralgorithm. Additionally, upon selection of regions of a given design forpremium finish, there are currently no tools for automaticallygenerating a secondary finish mask for the design. Thus, for every printdesign offered by a vendor, the vendor must expend resources designingone or more associated premium finish masks for that particular design.The design of a premium finish mask is therefore typically performed bya human graphics designer, often the same designer who created the printdesign. It would therefore be desirable to have automated toolsavailable that would automatically extract regions of a given design forpremium finish based on a selected area of the design image, and thatwould automatically generate one or more premium finish masks specificto the given design in order to minimize the amount of human designertime expended on the creation of a particular design without diminishingthe quality of the end design and premium finish aesthetics.

Additionally, often customers of a retail printed products provider maywish to provide an image to be printed, and to apply a premium finish(e.g., foil, raised print, etc.) to portions of the image to be appliedto the end printed product. No simple tools or techniques for indicatingwhich portions of the image are to be premium finished exist. Instead, atrained designer must hand create an appropriate premium finish maskspecific to the image the customer provided to ensure that the premiumfinish is applied in an aesthetically pleasing manner. Accordingly, itis difficult and expensive for end customers to add premium finish totheir own images. It would therefore be desirable to have availableweb-enabled tools that allow a customer to provide an image and toindicate regions of the image for application of premium finish. Itwould further be desirable to provide an intuitive user interface thatconforms to editing interfaces that customers are already used todealing with in other document editing applications.

SUMMARY

Various embodiments of the invention include automated systems, methodsand tools for automatically extracting and selecting portions of animage for application of a premium finish (i.e., a secondary finishseparately applied to a product before or after application of a primaryfinish). Embodiments of the invention also automatically generate apremium finish mask specific to the image based on the automatic (andoptionally, user-modified) segment selections. The system, method andtools require little or no human intervention.

In one embodiment, a method for generating by a computer system apremium finish mask image for applying a secondary finish to a productto be finished with a primary finish includes the steps of receiving animage comprising a plurality of pixels, generating a segmented imagehaving a plurality of pixels corresponding to the respective pluralityof pixels of the received image, the segmented image comprising discretesegments each comprising a subset of pixels, receiving a selection ofsegments designated for premium finish, and automatically generating amask image having first areas indicating non-premium finish areas andsecond areas indicating premium finish areas, the premium finish areascorresponding to the selection of segments designated for premiumfinish.

In an embodiment, the automated premium finish segment selection issubjected to a confidence level test to ensure that the automatedselection of segments corresponds well to a human value judgmentrelative to the aesthetics of premium finish applied to the selectedsegments in the final product.

In an embodiment, a graphical user interface provides user tools forallowing a user to modify an automatic segment selection

In an embodiment, a computerized system automatically receives an imageand generates an associated mask image for use in application of premiumfinish to a product.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating an exemplary method in accordancewith an embodiment of the invention;

FIG. 2 is a schematic block diagram of an exemplary system in whichembodiments of the invention may operate;

FIG. 3 is an example pixel map of a digital image illustrating theindividual pixels of the image;

FIG. 4A is an example original digital image;

FIG. 4B is an example of a smoothed version of the original image ofFIG. 4A;

FIG. 4C is an example of a color-segmented version of the smoothed imageof FIG. 4B, representing the color-segmented version of the originaldigital image in FIG. 4A;

FIG. 4D is an example of an outer bounding box used for selecting allforeground content of the image of FIG. 4A for designation as premiumfinish;

FIG. 4E is a gradient image of the selected segment image of FIG. 4D;

FIG. 4F is an example binary mask image created based on the selectedsegments of FIG. 4D;

FIG. 4G is an exemplary graphical user interface illustrating acustomizable design template for creating the design of the image ofFIG. 4A and the use of a FreeForm selection tool;

FIG. 4H is a gradient image of the design image corresponding to theselected segments as selected in FIG. 4G;

FIG. 4I is the graphical user interface of FIG. 8G illustrating the useof the Selection/Deselection marking tool;

FIG. 4J is a gradient image of the design image corresponding to theselected segments as selected in FIG. 4I;

FIG. 4K is the graphical user interface of FIGS. 4G and 41 illustratingthe use of a Rectangle selection tool;

FIG. 4L is a gradient image of the design image corresponding to theselected segments as selected in FIG. 4K;

FIG. 4M is an example binary mask image created based on the selectedsegments of FIG. 4L;

FIG. 5A is a simple example of an original image;

FIG. 5B is a corresponding color-segmented image of the original imageof FIG. 5A, illustrating introduction of new edges and disappearance oforiginal edges resulting from color-segmentation;

FIG. 6 is a flowchart illustrating an exemplary method for determining aconfidence level for an associated segment selection;

DETAILED DESCRIPTION

It will be understood that, while the discussion herein describes anembodiment of the invention in the field of preparation of customizedprinted materials having premium finish regions such as foil, gloss,raised print, etc., it will be understood that the invention is not solimited and could be readily employed in any embodiment involving thepresentation of an electronic image of any type of product wherein it isdesired to indicate one or more areas of the image that is to befinished using different finish than other areas of the image.

In accordance with certain aspects of the invention, described hereinare automated systems, methods and tools that automatically extractregions of a given design and automatically generate one or more premiumfinish masks specific to the given design while requiring little or nohuman intervention. In accordance with other aspects of the invention, aset of web-enabled tools are provided which allow a customer to providean image and to indicate regions of the image for application of premiumfinish. In accordance with other aspects of the invention, there areprovided simple intuitive user interfaces for indicating regions of animage to be finished with a secondary finish. In accordance with otheraspects of the invention, methods for capturing a designer's orcustomer's intent to premium finish specific parts of an image in themost intuitive, simple, reliable and useful way are described.

As is well known and understood in the art, color images displayed oncomputer monitors are comprised of many individual pixels with thedisplayed color of each individual pixel being the result of thecombination of the three colors red, green and blue (RGB). In a typicaldisplay system providing twenty-four bits of color information for eachpixel (eight bits per color component), red, green and blue are eachassigned an intensity value in the range from 0, representing no color,to 255, representing full intensity of that color. By varying thesethree intensity values, a large number of different colors can berepresented. Transparency is often defined through a separate channel(called the “alpha channel”) which includes a value that ranges between0 and 100%. The transparency channel associated with each pixel istypically allocated 8 bits, where alpha channel values range between 0(0%) and 255 (100%) representing the resulting proportional blending ofthe pixel of the image with the visible pixels in the layers below it.

In general, given an image from which to generate a correspondingsecondary finish mask, the image is segmented and then certain segmentsare selected for secondary finish, from which a corresponding mask imageis automatically generated. FIG. 1 is a flowchart illustrating thegeneral method. As a preliminary matter to discussion of the details ofthe method in FIG. 1, the boxes and shapes designating the processingsteps of the method are conventionally outlined using a solid line toindicate that the step, or at least an equivalent of the step, isperformed, and are otherwise conventionally outlines using a dashed lineto indicate that the step may or may not be performed but rather dependson the particular system and application to which it is applied.

Turning now in detail to FIG. 1, there is shown therein a method inwhich a processing system receives an original design image (step 1),processes the received design image to generate a segmented imagecomprising a plurality of segments (step 2), receives a selection ofsegments designated for premium finish (step 3), and generates a maskimage indicating the selected segments (step 4). The design image andthe mask image are used to manufacture the design on a product (step 5).In one implementation, a primary finish is applied, based on the designimage, to the product (step 6 a) and then a premium finish is appliedbased on the mask image (step 7 a). In an alternative embodiment, thepremium finish is applied first, based on the mask image (step 6 b),followed by application of the primary finish based on the design image(step 7 b).

As will be discussed in more detail hereinafter, in a preferredembodiment the original design image is segmented by first applying asmoothing filter which preserves strong edges (such as the well-knownmean-shift filtering technique) (step 2 a) followed by extractingcolor-based segment labels (for example, using connected componentlabeling based on color proximity) (step 2 b).

As will also be discussed in more detail hereinafter, the selection ofsegments designated for premium finish received in step 3 may beobtained in one of several ways. In a first embodiment, the system issimply instructed of the segments that are to be premium finished—thatis the system receives the selected parameters by a remote entity (step3 a). In an alternative embodiment, the system heuristically determinesa set of segments that are to be selected (also step 3 a). For example,the system may be configured to receive a selected area of the designimage (which can be the entire design (or one to a few pixels justinside the outer boundaries of the image) or a portion of the imagethereof), and the system may be configured to separate the foregroundfrom the background and to automatically select only the foregroundareas for premium finish. In another embodiment, a set of user tools canbe exposed to the user (step 3 b) to allow the user to individuallyselect and/or deselect individual segments.

In additional embodiments, the system and method may determine aselection confidence measure (step 8 a) and compare it against a minimumconfidence threshold (step 8 b). In an embodiment, if the value of theselection confidence measure meets or exceeds the minimum confidencethreshold, this indicates that the system is confident that the selectedset of segments reflect the intent of the user/designer to designatesaid segments as premium finish. In such case, it may be desirable todisplay a preview image of the final product containing the productcontaining the premium finish in the selected areas of the design (step9). If the value of the selection confidence measure does not meet theminimum confidence threshold, and/or optional in the case where theselection confidence measure does meet the minimum confidence threshold,the segmented image may be sent for human review and touch up tofinalize the selection of segments designated for premium finish.

FIG. 2 is a block diagram illustrating an exemplary system 100 in whichvarious embodiments of the invention may operate. As illustrated, one ormore servers 120 (only one shown) include one or more processors 121,program memory 122 which stores computer-readable instructions forprocessing by the processor(s) 121, data memory 126 for storing imagedata 128 such as image(s) 105 received from customers operating clientcomputer(s) 110 and segmented images and mask images associated with theimage(s) 105, and communication hardware 125 for communicating withremote devices such as client computer(s) 110 over a network 101 such asthe Internet. The program memory 122 includes program instructionsimplementing a premium finish mask generation tool 130, which mayinclude an image smoothing filter 131, a color segmentation engine 132,a premium finish segment selection engine 133, and a selectionconfidence calculator 134.

One or more client computer(s) 110 (only one shown) is conventionallyequipped with one or more processors 112, computer storage memory 113,114 for storing program instructions and data, respectively, andcommunication hardware 116 configured to connect the client computer 110to the server 120 via the network 101. The client 110 includes a display117 and input hardware 118 such as a keyboard, mouse, etc., and executesa browser 119 which allows the customer to navigate to a web site servedby the server 120 and displays web pages 127 served from the server 120on the display 117.

Memory 122, 126, 113, and 114 may be embodied in any one or morecomputer-readable storage media of one or more types, such as but notlimited to RAM, ROM, hard disk drives, optical drives, disk arrays,CD-ROMs, floppy disks, memory sticks, etc. Memory 122, 126, 113, and 114may include permanent storage, removable storage, and cache storage, andfurther may comprise one contiguous physical computer readable storagemedium, or may be distributed across multiple physical computer readablestorage media, which may include one or more different types of media.Data memory 126 may store web pages 127, typically in HTML or other Webbrowser renderable format to be served to client computers 110 andrendered and displayed in client browsers 119. Data memory 126 alsoincludes a content database 129 that stores content such as variouslayouts, patterns designs, color schemes, font schemes and otherinformation used by the server 120 to enable the creation and renderingof product templates and images. Co-owned U.S. Pat. No. 7,322,007entitled “Electronic Document Modification”, and U.S. Pat. PublicationNo. 2005/0075746 A1 entitled “Electronic Product Design”, each describesa Web-based document editing system and method using separatelyselectable layouts, designs, color schemes, and font schemes, and eachis hereby incorporated by reference in its entirety into thisapplication.

In preface to a more detailed discussion of the premium finish maskgeneration tool 130, which in one embodiment executes on the server 120(but could alternatively be executed on a stand-alone system such as theclient computer system 110), a brief discussion of digital image storageand display is useful. A digital image is composed of a 2-dimensionalgrid of pixels. For example, with reference to FIG. 3, a digital imagecomprises a number of pixels arranged in a rectangular grid. Each pixelis represented by its corresponding positional x- and y-coordinates inthe grid. For example, an m×n pixel image would be represented as mcolumns and n rows, with pixel (0, 0) conventionally referencing the topleft pixel of the image, and pixel (m−1, n−1) conventionally referencingthe bottom right pixel of the image. The number of pixels in the imagedefines the resolution of the image, where high-resolution images have ahigher density (and therefore number) of pixels than low-resolutionimages. The specification of each pixel depends on the type of image(grayscale vs. color, the type of color model used, and the number ofgray levels/colors). For example, in a grayscale image, each pixel maybe represented by a single brightness value ranging from 0 (nobrightness—i.e., completely dark) to 255 (full brightness—i.e.,completely light). In a color image, each pixel may be represented bymultiple values. For example, in an RGB color model, the color of eachpixel is defined by a corresponding Red value (ranging from 0-255), acorresponding Green value (ranging from 0-255), and a corresponding Bluevalue (ranging from 0-255).

A product vendor, a customer of a product vendor, or product designermay wish to generate a design, based on an image such as a photograph orgraphic, which may be applied to a product, where the designincorporates one or more regions of premium finish. In more advancedapplications, it may be desirable to allow a customer to upload an imageto be converted to a design and to designate certain regions of thedesign (i.e., image segments) as premium finish regions. The premiumfinish regions are applied to the product via a separate process fromthe rest of the design. For example, the product may be a printedproduct such as a business card, and a customer may wish to have theircompany logo or customized text or even a photograph converted to designthat includes foiled areas. The addition of foil to a printed designrequires a printing process and a foiling process, which are separateprocesses. An example of a foiling process is stamping in which a metalplate engraved with an image of the areas to be foiled first strikes afoil film, causing the foil film to adhere to the engraved plate, andthen strikes the substrate (e.g., business card) to transfer the foil tothe desired areas of the design. Another foiling process is hot stamping(i.e., foil stamping with heat applied), while another type of foilingprocess involves applying adhesive ink in areas to be foiled, followedby application of a foil layer over the entire image, followed by foilremoval, such that foil remains on the areas to which it adheres to theglue. Clearly, each foil process is a separate process from the printingprocess, and requires different input—namely the mask image indicatingthe areas for application of foil. Other types of premium finish alsorequire different and separate processes. Embossment, for example, istypically performed by compressing a substrate (e.g., business cardstock) between two die engraved with an image representing areas wherethe substrate is to be embossed—one die having raised areas where theimage is to be embossed and the other having recessed areas where theimage is to be embossed. Clearly, the embossment process is a completelyseparate process than the printing of the design on the substrate. Ingeneral, then, application of a premium finish requires separateprocessing from the application of the primary finish, and the premiumfinish processing utilizes the premium finish mask image as instructionof where the premium finish should be deposited/applied to the product.

When a premium finish design is to be generated based on an image, areasof the image must be mapped to discrete segments of the image. In oneembodiment, the problem of separating the image into segments and thendetermining which segments to select for premium finish is treated atleast in part by determining and separating background regions fromforeground regions of the image. In order to handle all types of imagecontent, the background/foreground separation is performed for thegeneral case (i.e., distinguishing between content in the foregroundversus a multicolor background), rather than only from a single-colorbackground.

In a preferred embodiment, segments of the image are extracted by firstsmoothing the image using a smoothing filter 131 (see FIG. 2) thatpreserves strong edges yet removes small details by smoothing orremoving weaker edges. In an embodiment, the smoothing filter 131 isimplemented as a mean-shift filter, which is well-known in the art ofimage processing. Smoothing using a filter such as mean shift filteringis important because it removes less important edges, thereby reducingthe number of segments that one might otherwise get after applying colorsegmentation to the original image. Instead, by applying the smoothingfilter, the strong boundaries remain but the small detail is removed andweak edges are smoothed into areas of gradual color change. Aftersubsequently applying color segmentation on the smoothed image, theresulting number of segments is reduced from what it would otherwise be,which makes segment selection more manageable.

The image smoothing filter 131 may be implemented using otheredge-preserving smoothing techniques in place of mean shift filtering.For example, an anisotropic filter may be applied instead, to sharpendetected boundaries by smoothing only at angles oblique to the detectededges where strong edges are detected and to smooth in all directions inareas where no (or weak) edges are detected. It is possible that stillother smoothing algorithms can be applied without departing from thescope of the invention so long as the stronger edges in the image arepreserved and the resulting smoothed image operates to assist in thefinal segmentation to reduce the final number of color segments to amanageable number of segments (which can be independently selected orunselected) while simultaneously preserving significant image features.

Once the original image is smoothed by the smoothing filter 131, theimage segmentation engine 132 (see FIG. 2) performs the finalsegmentation by processing the smoothed image into a color-segmentedimage. In an embodiment, the image segmentation engine is implemented asan image processor that performs a technique referred to as connectedcomponent labeling based on color proximity (can be in RGB, LAB or anyother color space producing desired results). In connected componentlabeling, regions or “blobs” of similar pixels are identified usinggraph theory. In particular, subsets of connected pixels are identifiedand labeled as “connected” based on the formation of a set of graphscomprising a set of vertices and connecting edges. Connected componentlabeling is one technique for identifying and separating backgrounddetail from foreground detail in an image, and identifying and labelingforeground regions as selectable for premium finishing.Connected-component labeling is a well-known technique in computervisioning for extracting regions or blobs in an image.

Below is a section of pseudo-code that exemplifies one implementation ofthe image segmentation engine 132 based on connected component labelingbased on color proximity:

First pass: pixel labeling Input: image with edge-preserving smoothingapplied (2-dimensional array colors[height, width]) Output: pixel labeldata (2-dimensional array labels[height, width])process_image(in:colors, out:labels)  current_label = 100 //initializelabel  for y = 0 to height-1 //loop through pixel rows  for x = 0 towidth-1 //loop through pixel columns   if labels[y, x] = 0 then  //ifpixel is not labeled yet   color = colors[y, x]  //remember the color of current pixel   process_pixel(color, y, x, labels)  //process currentpixel   current_label++  //generate next label   end  end  end endprocess_pixel(in:colors, in:color, in:y, in:x, in:current_label,in/out:labels)  labels[y, x] = current label //label the current pixelwith the current label  //loop through all immediate neighbors of thecurrent pixel  for dy = −1 to +1  for dx = −1 to +1   neighbor_y = y +dy   neighbor_x = x + dx   if (neigbor_y, neighbor_x) is not currentpixel (y, x) and not outside image then   neighbor_color = colors[y, x]  if |color − neighbor_color| < threshold then   //check whether colorsare similar enough   process_pixel(colors, color, neighbor_y,neighbor_x, current_label, labels)   //recursively process the neighborpixel   end  end  end end

Second Pass: Calculating Colors of Segments

-   -   Input: image with edge-preserving smoothing applied        (2-dimensional array colors[height, width])    -   Input: pixel label data (2-dimensional array labels[height,        width])    -   Output: segmented image (2-dimensional array segments[height,        width])    -   color_segments(in:colors, in:labels, out:segments)    -   for each label        -   for each pixel marked by that label, replace its color with            a color which is a simple average of all pixels with same            label and record in the segments array        -   end        -   end

The result of the above algorithm is a segmented image labeled based oncolors. Each segment comprises an exclusive subset of pixels set to thesame color.

In an alternative embodiment, separation of the foreground regions fromthe background regions may be performed by one of several availabletools, for example the “Remove Background” tool available in MS Office2010, available from Microsoft Corporation, which implements a versionof an algorithm referred to as the “Grab Cut” algorithm, which isdescribed in detail in C. Rother, V. Kolmogorov, and A. Blake, “GrabCut:Interactive foreground extraction using iterated graph cuts”, ACM Trans.Graph., vol. 23, pp. 309-314, 2004, and is hereby incorporated byreference for all that it teaches. Such products work well onphotographic images but are not effective at detecting holes in theforeground and understanding that the holes are part of the background.This can be a problem when important areas of the image contain text:the amount of work required to remove holes from line-art type images isvery large—one would need to unselect every hole inside each individualletter/character when text present, and this can be time-consuming.Furthermore, the currently available solutions are not very predictable:a small change in the position or size of the selection rectangle cancause abrupt changes in what gets removed (i.e., the algorithm is notlocal).

Other methods of performing color segmentation of the smoothed image maybe applied in place of connected component labeling based on colorproximity without departing from the scope of the invention. Forexample, in an alternative embodiment to separation of the foregroundareas from the background areas of the image, it may instead beconvenient to segment the areas of the image based on colors containedin the image itself. In general, a color segmented image is generated byiteratively merging pixel colors based on pixel color similarity andedge strength until the image contains only a predetermined number ofcolors or predetermined number of color segments. This process is knownin the image processing art as “color reduction”. During a colorreduction process, all of the pixels in the image are classified and setto the nearest one of a reduced number of colors. The reduced set ofcolors can be determined by iterative color merging, or by identifyingthe dominant or most frequently used colors in the image.

Upon production of a segmented image by the segment selection engine133, segments are designated for application of premium finish. Theselection of the set of segments designated for premium finish may varydepending on the particular use of the system. In one embodiment, asindicated at step 3 a in FIG. 1, the system automatically selects one ormore of the labeled segments and designates the selected segments forpremium finish. There are multiple possible alternatives for choosingwhich segments to automatically select. In an embodiment, all foregroundsegments are automatically selected for premium finish. This can beachieved by automatically selecting a bounding box having a perimeterjust inside (by one or a small few pixels) the outer edges of the image,and then assuming that any labeled segment which the bounding boxoverlaps is background and any labeled segment that is fully inside thebounding box (i.e., segments which the bounding box does not cross oroverlap) is a foreground segment. Also, any segment having color similar(within a fixed threshold, for example) to the color of the backgroundis classified as background. This can assist in identifying andclassifying holes in the foreground segments as background as well;however, when the bounding box passes through multiple segments due to amulti-color background, determining holes becomes much more difficultand is typically not able to be done automatically.

Referring now to FIG. 4A, there is shown an exemplary original image 40,which is an image of content to be applied to business card stock togenerate a business card product. FIG. 4B illustrates an exemplarycorresponding smoothed image 41, and FIG. 4C shows the correspondingcolor-segmented image 42 resulting from color segmentation of thesmoothed image of FIG. 4B. FIG. 4D illustrates a highlighted image 43showing the image 40 with a bounding box 44 near the edges that selectsthe entirety of the image content (less the outermost one to fewrows/columns of pixels of content—see exploded view at 44 a). Asillustrated, since the bounding box 44 overlaps segment S0, segment S0is thus identified as a background segment. All segments having asimilar color (within a fixed threshold, for example) as segment S0 arealso identified as background. For example, segment S2, corresponding tothe hole inside the letter “A” is identified as background. It will benoted that many applications of the present invention will be tomanufacturing products by applying designs that include text. Therefore,hole detection may be very important to automating the mask generationfor premium finishes. Accordingly, additional techniques for ensuringaccurate segmentation may be applied. Segments which correspond to text(and corresponding holes contained in the text) can be detected, forexample using the methods and systems described in patent publicationUS20130044945 A1 published Feb. 21, 2013 and entitled “Method and Systemfor Detecting Text in Raster Images” (filed as U.S. patent applicationSer. No. 13/210,184 on Aug. 15, 2011), which is hereby incorporated byreference herein for all that it teaches. All remaining segments thatare not identified as background are designated as foreground segments.Since all foreground segments fully contained within the bounding box 44are automatically selected for premium finish, they are highlighted inFIG. 4D to indicate they are selected. It will be noted that FIG. 4D(i.e., the highlighted version of FIG. 4A highlighting the areasdesignated for premium finish) is shown herein merely for instructivepurposes in demonstrating how the segments are selected relative to thebounding box 44. It will be noted, however, that image 43 is not likelyto be generated as such in practice, as the bounding box 44 and thesegment selections will generally be represented in the system as a datastructure or programmed equation or other internal representationsstored in memory.

FIG. 4E is a gradient image 45 of the selected segments, which, asdiscussed hereinafter indicates how well the selected segmentscorrespond to areas with strong edges in the original image. FIG. 4Fillustrates a binary mask image 46 corresponding to the segments of FIG.4C fully contained within the bounding box 44.

For some systems, selecting the entire foreground portion of the imagefor premium finish may be what is desired. In other systems, however,premium finish is used more for highlighting certain features in theimage, and thus it would not be appropriate and/or desired to applypremium finish to the entirety of the foreground regions of the image.One of the goals of the systems, methods and tools presented herein isto make premium finish mask creation as simple as possible for bothdesigners and end-user customers of products incorporating applieddesigns. According to one aspect, this is achieved by simply automatingthe selection of the segments for premium finish. However, should it bedesired to select less than the entire foreground content of the image,the determination of the segments to be premium finished can be quitedifficult due to the amount of detail in most photographic images.Accordingly, in some embodiments, as indicated in step 3 b of FIG. 1,one or more user tools may be provided to allow a user of the system toindividually and/or group-wise select/deselect segments of the image fordesignation as premium finish. Thus, it is useful to a designer orend-customer to have available simple intuitive segment selection toolsto hand touch (i.e., select and/or deselect) segments for designation aspremium finish. Accordingly, in some embodiments, the system may includeUser Selection/Deselection Tool(s) 135 (see FIG. 2).

In an embodiment the system and method includes tools and steps forexposing user tools on a user's (i.e., designer's or customer's) displayon a client computer system 110 (step 3 b in FIG. 1). FIG. 4G shows anexample graphical user interface 200 presented to a user during designof the business card product that is represented by the image 40 of FIG.4A. The graphical user interface 200 includes a work area 202 whichdisplays the current design. The design in the example includes textcorresponding to user-editable content in user input boxes 201 a-201 kreciting a company name 201 a, a company message 201 b, a name 201 c, ajob title 201 d, address lines 1, 2 and 3 (201 e, 201 f, 201 g), a phonenumber 201 h, a fax number 201 i, an email address 201 j, and a companywebsite URL (uniform resource locator) 201 k. The design also includesan image 220, which may be a digital photograph, a line art image, agraphical image, etc. In an embodiment, the design displayed in the workarea 202 is at least partially a What-You-See-Is-What-You-Get (WYSIWYG)display in terms of the content, placement, fonts, colors, of thecontent. However, since it is difficult to accurately represent premiumfinishes such as foil, gloss, and other shiny or textured material on acomputer display screen due to the scattering of light on such surfaces,the areas of the image designated for premium finish do not appear aspremium finish in the work area 202. Optionally, a full rendering of thedesign in the work area with simulation of light across the selectedareas of premium finish may be generated and displayed as a previewimage 210 a to give the user an accurate depiction of how the finalproduct will appear when physically manufactured. One way for simulatinga preview image of the finished product with premium finish applied isdescribed in detail in U.S. patent application Ser. No. 13/210,184,filed Aug. 15, 2011, and entitled “Method and System for Detecting Textin Raster Images”, and is incorporated by reference herein for all thatit teaches.

The graphical user interface 200 may include one or more user tools 204,205, 206 to individually select and/or deselect segments of the design(including segments of the image contained in image container 220) toinclude the user-selected segments or exclude the user-deselectedsegments from the set of segments designated for premium finish.

In the illustrative embodiment, the user segment selection/deselectiontools include a Rectangle Selection tool 204, a FreeForm Selection tool205, and a Selection/Deselection marker tool 206. FIGS. 4G, 4J and 4Mtogether illustrate the use of user tools to select and fine-tune apremium finish selection.

In FIG. 4G, the user has selected the FreeForm tool 205 (shown ashighlighted) to draw, using the cursor 203, a FreeForm selection shape232 circumscribing a portion of the image contained in the imagecontainer 220. In an embodiment, the FreeForm selection shape 232instructs the system to select all segments in the correspondingsegmented image (FIG. 4C) that are fully contained within the footprintof the FreeForm selection shape 232. FIG. 4H is a gradient image withthe selected segments 240 highlighted and with the FreeForm selectionshape 232 overlaid.

The FreeForm drawing may not always allow the level of accuracy that auser intends. For example, due to low resolution of the match of cursormovement to pixel position, it may be that occasionally some segmentsthat the user does not intend to premium finish get unintentionallyselected. For example, as illustrated in the gradient image in FIG. 4H,the selected segments, as indicated by the highlighted portion 240,include portions 240 a of the image background surrounding the flowerpetals. However, it is likely that the user intended to premium finishonly the “flower petals” content in the image. For this reason, the usertools may also include a fine-tune Selection/Deselection marker tool 206which allows a user to mark small areas for deselection and/orselection.

In FIG. 4I, the user has selected the Selection/Deselection marker tool206 (shown as highlighted) to mark areas 233 inside the FreeFormselection area 232 to indicate that such areas are background areas andnot to be included in the premium finish selection. As illustrated, thegradient image, shown in FIG. 4J, has been updated to show that only the“flower petal” portions are highlighted, indicating that only thesegments corresponding to the “flower petal” portions are designated forpremium finish.

FIG. 4K illustrates the use of the Rectangle selection tool 204. In FIG.4K the user has used a mouse to select the Rectangle selection tool 204(shown as highlighted) to draw, using cursor 203, a selection rectangle231 around portions of the text to indicate that the selected text is tobe included in the premium finish selection. As illustrated, thegradient image, shown in FIG. 4L, has been updated to show that the“flower petal” segments and the text circumscribed by the selectionRectangle 231 are highlighted and are designated for premium finish.

When the user is satisfied with the premium finish selection, the usercan save the design by clicking the save button 211 using the cursor203. Furthermore, the user can instruct the system to generate a premiumfinish mask corresponding to the design by clicking on the Create Maskbutton 212 using the cursor 203. Alternatively, the mask creation may beperformed automatically by the system when the user saves the design.FIG. 4M shows an example premium finish mask (i.e., a binary image)generated by the system which corresponds to the final design shown inFIG. 4K.

It is to be noted that the selection Rectangle 231, the selectionFreeForm shape 232, and the selection/deselection markers 232, are notpart of the design itself but are rendered on the user's display on aseparate layer over the rendered design in work area 202 merely to allowthe user to view where the user has placed the selection areas. Inimplementation, once a portion (or all) of the image is selected by aselection Rectangle or a selection FreeForm shape, the system analyzesthe boundary of the selection shape and only selects segments, based onthe color-segmented version (FIG. 3C) of the original image, which arefully inside the contour of the selection This can be performed by firstselecting all segments having any portion within the boundaries for theselection shape, and then determining and deselecting each segment thathas a pixel that corresponds to the position of any pixel of theselection boundary in the selection boundary layer. A similar processcan be performed for the selection/deselection indicators232—determining and selecting/deselecting each segment that has a pixelthat corresponds to the position of any pixel of the correspondingselection/deselection indicators 232.

In general, the selection engine operates to select for premium finishall segments within selection area bounded by a selection boundary. Inembodiments which perform automatic selection of the entire foreground,for example in systems where user tools may not be offered, theautomatic selection is achieved by automatically setting the selectionarea boundary to the outer pixels (or near the outer edges) of theimage, as illustrated at 44 a in FIG. 4D. Embodiments which allowgroup-wise segment selection by the user via one or more area selectiontools in a graphical user interface such as illustrated in FIGS. 4G, 4Iand 4K, the selection area is the area of the image inside the selectiontool boundaries (e.g., the selection Rectangle, the selection FreeFormshape). That is, when an area of the image is selected for premiumfinish, the system assumes that the boundary of the area selector (e.g.,the automatically placed Boundary Box 44, the Selection Rectangle box231, the FreeForm selection shape 232) is drawn over/through backgroundportions of the image—therefore, the remaining segments inside theboundaries of the area selector(s) are considered to be foreground andis/are therefore designated as selected for premium finish. Embodimentswhich allow fine-tuning of segment selection via one or more areaselection tools such as the Selection/Deselection marker tool 206 in agraphical user interface such as illustrated in FIG. 4I, any segmentthat has any pixel that corresponds to the position of any pixel of aselection mark (made by the Selection/Deselection marker tool 206) isselected, and any segment that has any pixel that corresponds to theposition of any pixel of a deselection mark (made by the tool 206) isdeselected (or conversely selected if already deselected).

Important to the ease of use of the present invention is that thesegments automatically selected for premium finish accurately andaesthetically reflect what the user or other human would believe is adesirable and pleasing application of the premium finish. In otherwords, is the premium finish on the final product going to be applied inareas that enhance the overall appearance of the design, and does itmake sense to an average consumer of the product?

In this regard, in a preferred embodiment, the system 100, and inparticular the premium finish mask generation tool 130, includes aselection confidence calculator 134 (see FIG. 2). One of the goals ofthe systems, methods and tools presented herein is to ensure that thepremium finish mask created for a particular image will result in anaesthetically pleasing design that accurately reflects what the designeror customer does or would intend. That is, when a certain area or all ofan image is selected, either automatically or manually by a designer orcustomer, it is important that the segments selected by the system areactually the segments that the designer or customer actually does orwould intend to finish with a premium finish.

The system is thus preferably equipped with a selection confidencecalculator 134 which calculates a value that corresponds to a level ofconfidence that corresponds well to a human judgment about the qualityof the selection of segments understood by the system as selected forpremium finish. In an embodiment, the selection confidence calculator134 generates a number (for example, but not by way of limitation, inthe range [0 . . . 1]), referred to herein as the “SelectionConfidence”, which in one embodiment reflects the degree of separationof foreground from background. In an embodiment, after the segmentedimage is generated, the selection confidence calculator 134 calculatesthe gradient of the original image (FIG. 4A) and the final selectedsegments (FIG. 4K) and calculates a color gradient match along thecontours separating selected segments from non-selected segments. Ingeneral, if the gradient match along the contours separating selectedsegments from the non-selected segments is, for the most part, high,this indicates that the particular contour representing a transitionbetween premium finish and non-premium finish follows along strong edgesin the original image, which is a strong indicator that the selectedsegment is intended to be selected—that is, the Selection Confidencewould be high. Conversely, if the gradient match along the contoursseparating selected segments from the non-selected segments is for themost part low, this indicates that the particular contour representing atransition between premium finish and non-premium finish follows a paththrough smooth areas or along weak edges in the original image, and theSelection Confidence would therefore be low, indicating that one or moreselected segments are not likely to be intended for selection.

The human eye is sensitive to image features such as edges. In imageprocessing (by computers), image features are most easily extracted bycomputing the image gradient. The gradient between two pixels representshow quickly the color is changing and includes a magnitude component anda directional component. For a grayscale image of size M×N, each pixelat coordinate (x,y), where x ranges from [0 . . . M−1] and y ranges from0 to N−1, defined by a function ƒ(x,y), the gradient off at coordinates(x,y) is defined as the two-dimensional column vector

${\nabla f} = {\begin{bmatrix}G_{x} \\G_{y}\end{bmatrix} = {\begin{bmatrix}\frac{\partial f}{\partial x} \\\frac{\partial f}{\partial y}\end{bmatrix}.}}$

The magnitude of the vector is defined as:

${\nabla f} = {{{mag}\left( {\nabla f} \right)} = {\left\lbrack {G_{x}^{2} + G_{y}^{2}} \right\rbrack^{1/2} = {\left\lbrack {\left( \frac{\partial f}{\partial x} \right)^{2} + \left( \frac{\partial f}{\partial y} \right)^{2}} \right\rbrack^{1/2}.}}}$

For a color image of size M×N, each pixel at coordinate (x,y) is definedby a vector having a Red component value R(x,y), a Green component valueG(x,y), and a Blue component value B(x,y). The pixel can be notated by avector of color components as:

${c\left( {x,y} \right)} = {\begin{bmatrix}{R\left( {x,y} \right)} \\{G\left( {x,y} \right)} \\{B\left( {x,y} \right)}\end{bmatrix}.}$

Defining r, g, and b as unit vectors along the R, G, and B axis of theRGB color space, we can define the vectors

$u = {{\frac{\partial R}{\partial x}r} + {\frac{\partial G}{\partial x}g} + {\frac{\partial B}{\partial x}b}}$

and

$v = {{\frac{\partial R}{\partial y}r} + {\frac{\partial G}{\partial y}g} + {\frac{\partial B}{\partial y}b}}$

and then define the terms g_(xx), g_(yy), and g_(xy) in terms of the dotproduct of these vectors, as follows:

$g_{xx} = {{u \cdot u} = {{\frac{\partial R}{\partial x}}^{2} + {\frac{\partial G}{\partial x}}^{2} + {\frac{\partial B}{\partial x}}^{2}}}$$g_{yy} = {{v \cdot v} = {{\frac{\partial R}{\partial y}}^{2} + {\frac{\partial G}{\partial y}}^{2} + {\frac{\partial B}{\partial y}}^{2}}}$$g_{xy} = {{u \cdot v} = {{\frac{\partial R}{\partial x}\frac{\partial R}{\partial y}} + {\frac{\partial G}{\partial x}\frac{\partial G}{\partial y}} + {\frac{\partial B}{\partial x}\frac{\partial B}{\partial y}}}}$

The direction of the gradient of the vector C at any point (x,y), isgiven by:

$\theta = {\frac{1}{2}{\tan^{- 1}\left\lbrack \frac{2g_{xy}}{g_{xx} - g_{yy}} \right\rbrack}}$

and the magnitude of the rate of change at (x,y), in the direction of θ,is given by:

${F(\theta)} = {\left\{ {\frac{1}{2}\left\lbrack {\left( {g_{xx} + g_{yy}} \right) + {\left( {g_{xx} - g_{yy}} \right)\cos\; 2\theta} + {2\; g_{xy}\sin\; 2\theta}} \right\rbrack} \right\}^{1/2}.}$

FIGS. 5A and 5B provide a simple illustration of how gradient matchingcan be used to determine segment selection confidence. FIG. 5A shows aregion 21 of an image having a color that gradually changes across theregion (e.g., from left to center and from center to right). Since thecolor transition is smooth and gradual, there are no visible edgefeatures between the left, center, and right portions of the region 21.When color segmentation is performed, however, as illustrated in FIG.5B, the far left and far right portions 21 a, 21 c of the region 21 maybe classified as a different color than the center portion 21 b of theregion. Pixels falling within the region 21 will be classified one wayor another according to color range thresholds. Pixels at the far left21 a and far right 21 c of the region 21 have a color value which fallswithin the range of a first color and are therefore classified as, andset to, the first color. Pixels closer to the center portion 21 b of theregion 21 have a color which falls within the range of a second colorand are therefore classified as, and set to, the second color. Theclassification of all pixels in the region 21 into only two (or a smallfew) discrete colors therefore results in the fracturing of continuouscolor transitions into two or more discrete colored regions, introducing(i.e., inducing) new edges 22 a, 22 b that were not present in theoriginal image.

When the gradient is calculated across the image 30, the newlyintroduced edges 22 a and 22 b are easily detectable. A comparison ofthe gradient calculated across image 20 and across image 30 will revealthe newly introduced edges 22 a, 22 b and this information can be usedto inform the Selection Confidence calculator 134 that the selection ofone or the other (but not both) of the segments 21 a and 21 b may resultin an undesired premium finish result. That is, if one or the other ofthe segments 21 a and 21 b is selected for premium finish, the premiumfinish will only serve to highlight the newly introduce edge(s) 21 a, 21b which did not exist in the original image. This can adversely affectthe aesthetics of the finished product.

Color segmentation can also result in the elimination of originalboundaries. For example, referring again to FIG. 5A, an image may haveone or more regions 24, 25 of different colors, separated by distinctboundaries or edges 26. The different colors of the regions 24 and 25,however, may each fall within a color range which maps to the samesingle color for purposes of color reduction. In this case, as shown inFIG. 5B, the sub-regions 24 and 25 are reduced to a single mono-colorregion 28, eliminating the original boundary 26 (see FIG. 5A) dividingthem. Depending on how significant the eliminated feature was, this mayor may not be acceptable in the final appearance of the embroideredimage. In this case, if the segment 28 is selected for premium finish,then the entire region including the region 25 which was originallydistinguished via original edges 26, will be finished with premiumfinish. Thus, the shape 25 may disappear in the final product,especially if the premium finish is applied on top of the primaryfinish. Again, gradient match can be used to detect such a condition. Inthis case, it is the original image which contains edges that do notappear in the segmented image. Calculating the pixel match across theselected segments between the original image and the selected segmentimage will reveal lost edges.

Referring now to FIG. 4L, this image shows the gradient image of thesegmented image with final segment selections of FIG. 4K. (Note: thehighlights are not included in the actual gradient image and areincluded only for purposes of indicating which segments are selected.)As illustrated, (1) pixels in regions of constant color value map to azero value in the gradient image (and hence the corresponding gradientpixel is completely dark), (2) pixels corresponding to the onset of acolor value step (i.e., an edge) or ramp map to non-zero values in thegradient image, and (3) pixels corresponding to areas of colortransition (i.e., more gradual color change) map to proportionalnon-zero values in the gradient image.

The gradient image of the original image of FIG. 4A is alsogenerated/calculated (but not shown herein), and the pixel values of theoriginal image gradient and the selected segment gradient are pixel-wisecompared along the contours between the selected and non-selectedsegments. The more differences there are between the correspondingpixels, the lower the gradient match along these contours, and the lowerthe Selection Confidence value. In general, it has been found that agradient match of at least approximately 90% is desirable when testedagainst human judgment. When a threshold is used to filter highconfidence selections from low confidence selections, an image passes aconfidence test (or returns a relatively high Selection Confidencevalue) when the gradient match is above the gradient match threshold,and conversely fails the confidence test (or returns a relatively lowSelection Confidence value) when the gradient match is below thegradient match threshold. For example, if there is a 95% match in pixelvalues along the contours between selected and non-selected segmentsbetween the gradient of the original image and the gradient image of theselected segment image (FIG. 4K), and the threshold is set to 90%, thenthe image passes the selection confidence test. If, however, thethreshold is set to 98%, then the selection would fail the selectionconfidence test.

FIG. 6 shows an example methodology followed by an exemplary SelectionConfidence calculator 130. As illustrated, the Selection Confidencecalculator 130 receives the original image (step 61) and the segmentedimage (step 62). The Selection Confidence calculator 130 generates agradient image corresponding to each of the original image and thesegmented image is generated (i.e., gradient original image (step 63)and a gradient segmented image (step 64)). The Selection Confidencecalculator 130 compares the selection area (i.e., the pixelscorresponding to pixels inside the bounding box, selection Rectangle,selection FreeForm shape, etc.) of the gradient original image to thegradient segmented image (step 65) to calculate how well the gradientscorresponding to the contours between selected and non-selected segmentsin the segmented image match the gradients of the corresponding areas inthe original image. In an embodiment, each pixel in the selection areaof the gradient original image is compared to its corresponding pixel inthe gradient segmented image, keeping track of the number of pixels thathave differing values. The gradient match is the number of matchingpixels over the total number of pixels in the selection area. TheSelection Confidence calculator 130 generates a Selection Confidencevalue corresponding to the calculated gradient match in the selectionarea. The Selection Confidence value represents how confident one mightbe that the selected set of segments in the selection area correspondwell to corresponding areas in the selection area of the originaldesign.

Returning to FIG. 1, once the Selection Confidence is determined in step8 a, it is evaluated against a minimum confidence threshold in step 8 b.As discussed above, the minimum confidence threshold may be set by thesystem designers, but is generally preferred to be at least 90% in orderto ensure close match between the selected segments and correspondingareas of the original image. In an embodiment, the Selection Confidenceis used to determine whether or not to present a preview image of thefinished product (step 9).

In an embodiment, in optional step 10 a the design image and segmentselections associated with the original design are sent to a humanreviewer who will take a look at the selected segments, and touch up theselected segments in the case that the system-generated segmentselection over-included or under-included segments for premium finish.For example, the segmentation may include small detailed segments thatthe user would not be aware of and which may inadvertently get includedin the selection area and be considered foreground segment. While itwould be difficult for a computerized system to recognize that such asegment should not be included for premium finish, a human reviewerwould easily recognize that such segment(s) are not part of theforeground image and can remove the selection of such segments.

Step 10 a may be performed only when the Confidence Selection does notmeet the minimum confidence threshold, or may alternatively be performedeven when the Confidence Selection meets the threshold.

In an embodiment, it may be desired to present a preview image of thefinished product, for example using the techniques described in U.S.patent application Ser. No. 13/973,396, filed Aug. 22, 2013, andentitled “Methods and Systems for Simulating Areas of Texture ofPhysical Product on Electronic Display”, and is incorporated byreference herein for all that it teaches, in order to show theuser/designer what the final product will look like with the premiumfinish applied.

In a system which sends designs and segment selections to a humanreviewer for final touchup, and especially in cases where the SelectionConfidence is below the minimum confidence threshold such that the humanreviewer would be likely to make noticeable changes to the set ofsegments selected for premium finish, it may not be desirable to displaya preview image in such cases. Thus, in an embodiment, a preview imageof the finished product is displayed (step 9) only when the SelectionConfidence value meets or exceeds the minimum confidence threshold (forexample, 90% gradient match). If such embodiment, if the SelectionConfidence value does not meet the threshold, the preview image is notdisplayed.

In yet another embodiment, if the Selection Confidence value meets orexceeds the minimum confidence threshold, for example when the minimumconfidence threshold is set to a high value (such as, for example andnot limitation, 98%), the human review in step 10 a may be skipped.

Once the segments designated for premium finish are selected andfinalized, the system generates a mask image indicating the selectedsegments (step 4). In this regard, in an embodiment a mask image isgenerated from a design image based on the segment selections. In anembodiment, in the corresponding mask image, pixels corresponding to theselected segments of the received image are set to a first predeterminedRGBA (RGB plus Alpha) value, while pixels corresponding to the remainingportions of the image are set to a different predetermined RGBA value.In an embodiment, pixels corresponding to premium finish segments areset to a White RGB value, whereas pixels corresponding to non-premiumfinish segments are set to a Black RGB value. Other mask file formatsand other color schemes may alternatively be used.

Optionally, the original design image and corresponding mask image, andoptionally the segmented image from which the mask image is generated,are passed to a human reviewer for a final review prior to manufactureof the design on a product. For example, in a production environment,members of a human production support staff may review designs andcorresponding masks prior to manufacture to ensure that the mask imageenhances areas which make sense using human judgment and does notenhance areas of the image that would not make sense. If the humanreviewer judges that automatically generated mask image indicates areasof premium finish that do not make sense based on the original image, ordoes not include areas indicated for premium finish that the humanjudges should be included in the premium finish, the human reviewer maytouch up the mask image by adding or removing areas indicated forpremium finish (step 10 b).

In application, the image itself, or a design based on the image, ismanufactured on a product (step 5). In the illustrative embodiment shownin FIGS. 4A-4K, the image is a business card design and the businesscard design is manufactured on card stock to create a set of businesscards. In an embodiment, the primary finish is ink printed on the cardstock. In an embodiment, the premium finish is foil, gloss, raised(3-dimensional) print, etc.

The order that the primary finish and premium finish is applied can varydepending on the desired finished effect. In one embodiment, the designis applied using the primary finish first (step 6 a) and then premiumfinish is applied to areas of the product as indicated in the mask image(step 7 a). In an alternative embodiment, the premium finish is appliedfirst (step 6 b)—the mask image used in the application of premiumfinish to the manufactured product in areas of the card stock asindicated by the mask image. The primary finish is applied second (step7 b)—in the illustrative embodiment, for example, the design (asindicated in the design image) is then printed on the card stock (suchthat ink may be printed over the premium finish). It is to be noted thatthe segments designated as premium finish as described herein may bedesignated as such only for the purposes of premium finish maskgeneration. In general, the original design image will still be finishedin full using the primary finish, whereas the premium finish will beapplied only in areas indicated by the mask image. For example, in theembodiment illustrated in FIGS. 4A-4K, the business card design isprinted in full on the business card stock, and the premium finish areas(e.g., foil) is applied only in the areas indicated in the correspondingmask image. The design may be printed after first applying the premiumfinish so that portions of the printed design image appear on top of thepremium finish. Likewise, it is possible that for some more transparentpremium finishes, the printed design is printed before applying thepremium finish and portions of the printed design appear through thepremium finish. In other words, the segments designated for premiumfinish need not be, but can be, mutually exclusive of the segments notdesignated for premium finish.

The method disclosed in FIG. 1 can be performed for one of severalreasons. First, a user of the system may have actively initiated themethod by, for example, requesting a premium finish product andproviding or selecting the original image. In this use case, the usermay desire that the system automatically select the segments to becoated with premium finish, in which case the system 100 automaticallyselects all foreground segments of the image for premium finish.

In a second use case, the user actively requests a premium finishproduct and provides or selects the original image, but in this use casethe user is provided with a set of user tools (step 3 b) to allow theuser to select one or more areas of the design in which premium finishis desired by the user. In this use case, the system must detect theuser's selection and decode the user's selection to identify thesegments of the image on which to base the premium finish mask image. Inthis use case, the Selection Confidence may be calculated (step 8 a) toensure that the selected segments make sense according to human judgmentgiven the original image, and if the threshold is met, the mask isautomatically generated, and otherwise the image and segment selectionand/or mask image is sent to a human reviewer for touch-up (step 10 aand/or 10 b).

In a third use case, the user may provide a design image which isdesired to be applied to a physical product, and the system thenautomatically segments the image, automatically selects segments fordesignation as premium finish (step 3 b), and a selection confidenceengine calculates the Selection Confidence (step 8 a). Then, if theSelection Confidence value meets or exceeds a minimum confidence level(step 8 b), a preview image of the design with premium finish appliedaccording to the automatically selected segments, is generated and shownto the user (step 9) and a product upgrade to a premium finished versionof the product is offered to the user (step 11)—namely, the same productdesigned by the user but enhanced with the application of premiumfinish. If the user accepts the offer (determined in step 12), themethod continues as described.

It will be appreciated from the above discussion that the systems,methods and tools described herein allow a simple efficient way toquickly generate a mask image indicating premium finish areas of aproduct to be manufactured. While embodiments are described in thecontext of manufacturing mask images for applying premium finish (suchas foil, gloss, or other separately applied texture) to a business cardor other printed product, the invention is not so limited. Theprinciples described herein may be applied to the manufacture of anyproduct that requires a separate mask image for the application of asecond finish to a physical product.

Those of skill in the art will appreciate that the invented method andapparatus described and illustrated herein may be implemented insoftware, firmware or hardware, or any suitable combination thereof.Preferably, the method and apparatus are implemented in software, forpurposes of low cost and flexibility. Thus, those of skill in the artwill appreciate that the method and apparatus of the invention may beimplemented by a computer or microprocessor process in whichinstructions are executed, the instructions being stored for executionon a computer-readable medium and being executed by any suitableinstruction processor. Alternative embodiments are contemplated,however, and are within the spirit and scope of the invention.

What is claimed is:
 1. A computer implemented method for segmenting anddesignating, by a server computer, premium finish regions based on animage received from a remote user using a client device in communicationwith the server computer via a communications network, the methodcomprising: the server computer generating from the received image asegmented image having a plurality of pixels corresponding to therespective plurality of pixels of the received image, the segmentedimage comprising discrete segments each comprising a subset of pixels;the server computer processing the segmented image to classify one ormore of the discrete segments as foreground segments and one or more ofthe discrete segments as background segments by receiving a selectionarea boundary that corresponds to a selection area of the segmentedimage and classifying all segments that are fully contained within theselection area of the segmented image as foreground segments andclassifying all segments outside of or that overlap the selection areaof the segmented image as background segments; and the server computerdesignating the foreground segments as premium finish regions and thebackground segments as primary finish regions; and providing thedesignation of the premium finish regions to the client device via thecommunications network.
 2. The method of claim 1, further comprising:the server computer generating a mask image corresponding to thereceived image, the mask image having first areas indicating the primaryfinish regions and second areas indicating the premium finish regions;and providing the mask image as the designation of the premium finishregions to the client device via the communications network.
 3. Themethod of claim 1, the server computer generating the segmented image bysmoothing the received image while preserving strong edges, andgenerating a color-segmented image from the smoothed image throughextraction of segment labels based on color.
 4. The method of claim 3,the server computer smoothing the received image by applying mean-shiftfiltering to the received image to generate the smoothed image.
 5. Themethod of claim 4, the server computer generating the color-segmentedimage is generated by applying connected component labeling to thesmoothed image based on color proximity.